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The outbreak of coronavirus disease 2019 (COVID-19) drives most higher education 
systems in many countries to stop face-to-face learning. Accordingly, many univer- 
sities, including Jordan University of Science and Technology (JUST), changed the 
teaching method from face-to-face education to electronic learning from a distance. 
This research paper investigated the impact of the e-learning experience on the stu- 
dents during the spring semester of 2020 at JUST. It also explored how to predict 
students’ academic performances using e-learning data. Consequently, we collected 
students’ datasets from two resources: the center for e-learning and open educational 
resources and the admission and registration unit at the university. Five courses in the 
spring semester of 2020 were targeted. In addition, four regression machine learning 
algorithms had been used in this study to generate the predictions: random forest (RF), 
Bayesian ridge (BR), adaptive boosting (AdaBoost), and extreme gradient boosting 
(XGBoost). The results showed that the ensemble model for RF and XGBoost yielded 
the best performance. Finally, it is worth mentioning that among all the e-learning 


components and events, quiz events had a significant impact on predicting the stu- 
dent’s academic performance. Moreover, the paper shows that the activities between 
weeks 9 and 12 influenced students’ performances during the semester. 
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1. INTRODUCTION 

Electronic learning (e-learning) uses digital technologies, such as computers and smartphones, to 
empower people to access and learn information anywhere and anytime, depending mainly on the internet [I], 
[2]. Thus, e-learning helped educational institutions’ users (i.e., teachers and students) to acquire various skills 
and valuable knowledge. During the coronavirus disease 2019 (COVID-19) pandemic, traditional education 
(face-to-face) was switched to online and distance learning by many educational institutions worldwide [B]. 
The e-learning system facilitated the educational process by providing a set of electronic educational materials 
and the possibility of sharing them in various formats (i.e., pdf, power point slides, word document, audios, 
videos, and images) [2]. It also allows, improves, and increases the interaction between teachers and students 
[3]. Teachers can perform several activities, including writing quizzes, exams, assignments, uploading several 
materials for a specific course, and asking specific questions to be discussed [I]. 

Several researchers and educators in academic institutions in different countries are interested in ex- 
ploring the prediction methods of students’ academic performances based on the previous records [4]-[7]. They 
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are also motivated to determine the relationship between e-learning system usage and students’ performances. 
This relationship can help the academic institutions to identify the various crucial activities that affect the ed- 
ucational process [8-10]. This research paper examines the e-learning system datasets in Jordan University 
of Science and Technology (JUST) during the spring semester of 2021. We have collected the data from five 
courses at the JUST in the spring semester of 2019/2020. We have selected the courses with the most significant 
number of students representing different faculties. The datasets include files related to students’ performances: 
log, grades, and user files. The total number of students in this study is 4,874. The data are manually filtered 
and analyzed to determine the significant features. This study aims to: i) provide an e-learning dataset to tackle 
the lack of dataset availability; ii) predict students’ performances in different courses using different machine 
learning algorithms (RF, BR, AdaBoost, and XGBoost); iii) expose the relationship between e-learning usage 
and students’ performances; and iv) analyze the error in the mentioned models for each course. 

The research paper’s reminder sections proceed: section 2 provides insights into the proposed method- 
ology used in this paper in terms of dataset collection, dataset description, pre-processing, feature extraction, 
and the used regression models. Next, section 3 discusses the experimental results and analyzes the error for 
the best ensemble regression model, and sheds light on the feature importance. Finally, section 4 shows the 
conclusion of the research study. 


2. RESEARCH METHOD 

This section presents the architecture of the proposed methodology used in this paper to predict the 
students’ performance using e-learning data at the appearance time of COVID-19. The architecture of this paper 
methodology includes collecting the data from the ’Center for E-Learning and Open Educational Resources” 
and the ’Admission and Registration Unit” at JUST, then analyzing and filtering the data. It also includes 
extracting the essential features and using the regression machine learning algorithms to predict the students’ 
performance. Finally, another crucial step in this study is to find the correlation between e-learning events and 
grades to find the most dramatic events on the students’ performance. The following subsection explains each 
of the architecture components in detail. 


2.1. Data collection and feature extraction 

The five courses data was collected from two units at the Jordan University of Science and Technology 
in the spring semester of 2019/2020. We have selected the courses with the most significant number of students 
representing different faculties. The total number of students is 4,874. Each course has i) logs file that contains 
entry time for e-learning, events performed using specific e-learning components, description of what the users 
are doing in each entry, how to access the system, and the internet protocol (IP) address of the device; 11) users 
file that contains information about the students and teachers in terms of student ID, teacher ID, and e-mails; 
and iii) grades file for the students regarding quizzes, assignments, first exam, second exam, mid-exam, final 
exam, and final grades. The first two files are obtained from the Center for E-Learning and Open Educational 
Resources, while the last one is obtained from the Admission and Registration Unit. Table[I|shows the course 
name, course id, and the number of students in each course in the Spring semester 2020. It is worth mentioning 
that the registered students are from eleven faculties. 


Table 1. Five courses at spring semester 2019/2020 


Course name Course id No. students No. of entries No. components No. event 

General biology laboratory BT107 1,002 79,874 6 8 

Biochemistry CHEM262 612 79,877 7 9 

Computer skills CISO099 876 97,195 7 L 

General skills HSS129 1,784 234,554 6 8 

General physics PHY 103 600 79,697 9 15 
Total number 4,874 


After collecting the dataset, the students’ names and IDs were masked using a masking strategy in 
which each student had a unique number. Finally, the three files were merged by matching each student’s 
logs with his information and grades. The students’ logs files’ main contents are components and events. The 
components are the group of features in e-Learning that include different resources to track students’ progress. 
While the events are the activities performed by the users using the components. The data set was filtered based 
on the entries related to students only. Table[I] shows the number of entries, components, and events for only 
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students in the spring semester 2020. This filtering extracts the components and events related only to students 
to predict their performance using the e-learning system at COVID-19 time. 

After filtering the dataset, the features were extracted from log files of the five courses. These features 
are the student’s faculty, what the way used to access the e-learning system, whether through their computer 
or phone, what the activities and resources viewed for a course, the number of discussions for each student, 
number of submitted, started, viewed and summary viewed of quizzes, number of message sent between teach- 
ers and students or among students, number of assignments’ submissions, number of times that each student’s 
view his grades or grades’ reports, and number of times that each student has done different activities per week 
during the semester. The quiz component with its events had the highest impact on the student’s performance 
in the five courses: the quiz attempt started, and quiz attempt submitted have the highest correlation values 
as : 0.49, and 0.51 in BT107, 0.47, 0.37 in CHEM262, 0.65, and 0.65 in CIS099, 0.73, and 0.72 in HSS129, 
0.38 and 0.36 in PHY 103. Therefore, we noticed that the quiz events have the most affected because it is done 
frequently and interested by the students during the semester. Furthermore, it is noticed that the activities of 
weeks between nine and thirteen have the highest correlation values than other weeks because it is the period 
for the student to prepare for the various semester works exams and final exams. In BT107, CHEM262, and 
PHY 103 courses, the students did not conduct any activities on e-learning. This is because the nature of these 
courses is practical. Compared with CISO99 and HSS129 courses, the students need to open the e-Learning 
because these courses’ nature is not practical. 


2.2. Regression models 


This study conducted experiments to predict students’ performances based on the semester work 
grades (out of 50). The semester work contains activities based on the course’s nature, such as quizzes, as- 
signments, first exam, mid-exam, and second exam. Thus, the semester works grades are out of 50 based on 
activities accomplished in each course. This research paper experimented with four regression models to pre- 
dict students’ semester work out of 50. To train the machine learning models, we have divided the dataset of 
each course into 70% of the dataset as a training dataset and 30% as a testing dataset. Several regression and 
correlation metrics were used in the experiments to evaluate machine learning models. The root mean square 
error (RMSE) is used as a standard popular regression metric in the prediction process. The RMSE is 
calculated as the difference between the predicted and actual continuous values. We have listed the RMSE 
with other metrics for our analysis, such as mean absolute error (MAE) [12], mean absolute percentage error 
(MAPE) [13], [14], R square (R*) [15], scatter index (SI) [16], Pearson correlation, and Spearman correlation 
[17). 

The regression models used in the current study are the RF, XGBoost, BR, and AdaBoost regressor 
models. RF regression model is a supervised ensemble model of several decision trees generated for both 
regression and classification tasks, in which each decision tree produces a prediction [18]. The XGBoost 
regression model is an ensemble model developed to solve classification and regression problems [19]. It 
contains several weak learners to generate a single strong learner. The weak learners are gradient boosting 
decision trees, in which each tree is performed individually, produces individual predictions, and then combines 
these predictions to form a final model’ prediction. However, unlike AdaBoost, XGBoost used the gradient 
descent algorithm to minimize the error between actual and predicted results, increasing speed and performance 
{19}. The third model is AdaBoost regression model, which is one of the boosting algorithms that is developed 
for both classification and regression tasks [20]—[22]|. It is an ensemble algorithm used to fit weak learners and 
produce a single strong learner. The predictions of weak learners are combined under final prediction whether 
weighted majority vote for classification tasks or average prediction for regression tasks [20]—(22]. The last 
model is BR regression model . The Bayes theorem uses the independence assumptions between the features 
in the inner work to predict the results. It computes the possibilities for each class label and determines each 
input data to which of the labels belongs to it based on the maximum possibilities [23]. The BR estimator 
proposed to solve the regression task from a Bayesian perspective [24]. Bayesian regression techniques can 
be used to include regularization parameters in the estimation procedure that are tuned to the data at hand. 
The BR regression model estimates a probabilistic model of the regression problem. We also applied voting 
regression model, which is an ensemble model that uses different types of machine learning models (classifiers 
or regressors) as base learners to produce the final results 25], [26]. The final prediction of the voting model 
follows the wisdom of experts’ pattern, and the output is the class that has the most votes from all classifiers in 


classification tasks [25], [26]. 
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3. RESULT AND DISCUSSION 


This section presents the experimental setup of five models: AdaBoost, BR, RF, XGBoost and the 
proposed voting model. Also, it presents the experimental results of these models. The experimental setup of 
the machine learning regression models used for the targeted course courses are: 


— AdaBoost regression model We have experimented with different parameters of AdaBoost. Table 
shows the best results of the different parameters used for four AdaBoost experiments and related results 
for each courses in these experiments. It is clear that none of the models is considered the best model 
for all courses. For example, AdaBExp (1) has the best prediction results for the courses CHEM262, 
CIS099, and PHY 103. However, it has the worst prediction results for BT107 course. 

— Bayesian ridge regression model: Table [3] shows the different parameters used in best four BR experi- 
ments and related results for each courses in these experiments. It is obvious that there is no best model 
for all courses. 

— Random forest regression model: Table [4] shows the different parameters used in four RF experiments 
and related results for each courses in these experiments. Once again, no best or worst model for all 
courses. 

— XGBoost regression model Table [5] shows the different parameters used in four best XGBoost experi- 
ments and related results for each courses in these experiments. None of the experimented models can 
be considered the best or the worst model to predict the results for all courses. 

— Voting regression model Table|6|shows the different parameters used in six proposed Voting experiments 
and related results for each courses in these experiments. 


Table 2. Experimental setup and results for AdaBoost 


Experiments Parameters RMSE RMSE RMSE RMSE RMSE 
BT107 CHEM262- CISO99  HSS129  PHY103 
AdaBExp. (1) learning_rate=0.01, 5.36 2.71 6.36 2.54 6.42 


n_estimators=50, 
random_state=42 

AdaBExp. (2) _ learning_rate=0.0001, 5.29 2.71 6.46 2.50 6.56 
n_estimators=50, 
random_state=42 

AdaBExp. (3) learning rate=0.001, 5.33 2.79 6.48 2.49 6.53 
n_estimators=50, 
random_state=42 

AdaBExp. (4) _ learning_rate=0.0001, 5.30 2.76 6.41 2.49 6.56 
n_estimators=500, 
random_state=42 


Table 3. Experimental setup and results for BR 


Experiments Parameters RMSE RMSE RMSE RMSE RMSE 
BT107 CHEM262- CISO99. HSS129  PHY103 
BRExp. (1) alpha 1=4e-6, compute score=True, 5.49 3.18 6.54 3.48 7.78 


n iter=100, normalize=True, 
fit intercept=True 
BRExp. (2) alpha 1=4e-6, compute score=True, 6.36 3.54 7.31 6.02 7.79 
n iter=100, normalize=True, 
fit intercept=False 
BRExp. (3) alpha 1=4e-6, compute score=True, 5.47 3.16 6.55 3.47 7.10 
n iter=2, normalize=True, 
fit intercept=True 
BRExp. (4) alpha 1=4e-6, compute score=False, 5.45 3.14 6.57 3.8 6.89 
n iter=10, normalize=False, 
fit intercept=True 
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Table 4. Experimental setup and results for RF 
RMSE RMSE RMSE RMSE RMSE 
BT107 CHEM 262 CISO99 HSS129 PHY103 


Experiments Parameters 


random state=42, max features=log?2, 
n estimators= 1000, max depth=None 
RFExp. (2) random state=42,max features=auto, 541 261 6.25 215 6.48 

n estimators= 1000,max depth=None 
random state=42,max features=sqrt, 
n estimators=1000,max depth=None 
random state=42,max features=auto, 


RFExp. (4) sh estimatere=sOanan-depth=0 5.38 2.26 6.49 2.49 6.74 


RFExp. (1) 5.31 2.44 6.22 2.14 6.49 


RFExp. (3) 5.33 2.40 6.23 2.11 6.49 


Table 5. Experimental setup and results for XGBoost 
RMSE RMSE RMSE RMSE RMSE 


Eee eee BT107_ CHEM262 CIS099 HSS129 PHY103 
random state=42,colsample bytree=1.0, 
AGRE sD ST) eta=0.30,max depth=2,subsample= 0.9 ane co bet 208 ce 
random state=42,colsample bytree=0.9, 
XGBExp. (2) eto-0.10 ix depth=l-aubsample 0S 5.20 2.34 6.26 2.20 6.61 
random state=42,colsample bytree=1.0, 
SGBExp. (3) sti-1i05, mux depik=?. cubsample= 09 5.26 2.45 6.18 2.18 6.58 
random state=42,colsample bytree=0.9, 
XGBExp. (4) eo=0.10 tow depth=? sahexmple= 05 5.16 2.50 6.18 2.11 6.55 
Table 6. Experimental setup and results for voting models 
Experiments Base regressors RMSE RMSE RMSE RMSE RMSE 
Course BT107 Course CHEM262 Course CISO99 Course HSS129 Course PHY 103 
VotExp. (1) BRExp. (4) + AdaBExp. (4) 5.28 277 6.33 2.58 6.50 
VotExp. (2) RFExp. (1) + XGExp. (4) 5.20 2.41 6.16 2.08 6.48 
VotExp. (3) RFExp. (4) + XGExp. (2) 5.21 2.27 6.33 2.30 6.64 
VotExp. (4) RFExp. (3) + BRExp. (3) 5.29 2.59 6.25 2.33 6.48 
+ AdaBExp. (1) 
VotExp. (5) XGBExp. (4) + RFExp. (4) 5.21 2.48 6.27 2.29 6.45 
+ BRExp. (4) + AdaBExp. (2) 
VotExp. (6) XGBExp. (3) + RFExp. (3) 5.26 2.55 6.24 2.23 6.48 


+ BRExp. (3) + AdaBExp. (3) 


The proposed voting model VotExp(2) shows the best results in terms of MAE for all courses (except 
for PHY 103, it is the second best result). Also, The summation value for all the RMSE of all courses yield 
that VotExp(2) has the least summation value of the RMSE. Thus, our proposed model is the voting model 
for RFExp(1) and XGExp(4). Figure |1}shows the proposed voting model. Table [7] shows the results of the 
regression machine learning models for the five courses at the spring semester 2019/2020. Depending on the 
results in Table[7| the results of all courses are explained as: 


— BT107 course: the best regression model that gives the least RMSE value in this course is XGBoost 
model with 5.16; however, the voting model VotExp(2) has a descent value of RMSE 5.20. 


— CHEM262 course: the best regression model that gives the least value of RMSE in this course is the 
RFExp(4) with 2.26. The voting model VotExp(2)’s RMSE is 2.41. 


— CIS099 course: the best regression model that has the least RMSE value in this course is the voting 
model VotExp(2) with 6.16. 


— HSS129 course: the best regression model that provides the least value of RMSE is the voting model 
VotExp(2) 2.08. 


— PHY103 course: the regression model that has the least RMSE value in this course is the voting model 
VotExp(8) with 6.35, and the voting model VotExp(2) is not even close 6.48. 
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E-LearningJUST dataset 


XGBoost Model 
randomstate=42 colsamplebytree=0.9 
eta=0.10 maxdepth=2  subsample=0.9 


Random Forest Model 
randomstate=42 maxfeatures=log2, 
nestimators= 1000 §maxdepth=None 


Ensemble Model 
Voting Method 


Prediction Results 


Figure 1. Proposed voting model VotExp(2) 


Table 7. Results for VotExp(2) 


2 


Courses Models RMSE MAE R- 
RF 5.31 4.25 0.34 
BT107 XGB 5.16 4.14 0.37 
Voting 5.20 4.17 0.36 
RF 2.44 1.33 0.58 


CHEM262 XGB 2.50 1.37 0.55 
Voting 2.41 1.32 0.59 


RF 6.22 4.93 0.45 

CIS099 XGB 6.18 488 0.46 
Voting 6.16 4.88 0.46 

RF 2.14 1.25 0.81 


-HSS129 XGB 2.11 1.20 0.81 
Voting 2.08 1.20 0.82 

RF 6.49 5.19 O41 

PYS103 XGB 6.55 5.16 0.40 
Voting 6.48 5.12 0.41 


The current study relies on the semester works’ grades. Thus, it is worth illustrating the grades’ 
distribution for each course. The distribution is divided into four groups of ranges: [0, 24], [25, 37] [38, 41] , 
and [42, 50]. Table [8|shows the percentage of students in these groups of ranges. The key findings based on 
Table ]is that the CHEM262 and HSS129 courses have the maximum percentage of students in the [42, 50] 
group and the lowest rate in the [0, 24] group. As a result, we can deduce why the RMSE values are better 
in CHEM262 and HSS129. Finally, the students who used e-learning intensively and frequently received high 
grades, as shown by the high correlation between the semester work and the total events in all courses. 


Table 8. The percentages results of four groups for all courses 
Course ID [42,50] [38,41] [25,37] [0,24] 


BT107 41.62% 42.71% 12.28% 3.39% 
CHEM262 93.46% 3.43% 1.8% 1.31% 
CIS099 28.77% 42.35% 19.41% 9.47% 
HSS129 91.31% 5.89% 1.40% 1.40% 
PHY 103 30.50% 40% 22.17% 7.33% 
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4. CONCLUSION 


This research paper provides a pilot study to predict students’ performance using e-learning data in 
five courses (BT107, CHEM262, CIS0O99, HSS129, and PHY103) during the spring semester 2019/2020. We 
have chosen these courses from different departments with different faculties at JUST University. We have also 
selected the semester work grades because they included all the grades of activities and exams taken before 
presenting the final exam, and the students’ performance will most affect it. To predict the students’ perfor- 
mance, we used five machine learning algorithms with regression types: the RF, XGBoost, BR, AdaBoost, and 
voting models. The results shown in all courses, the voting regressor gave the higher values in three courses 
and the second-best results in two courses. We then studied the relationship between e-learning usage and the 
students’ performance based on the most affected performance events. Based on the results achieved, there is a 
high correlation between e-learning usage and the students’ performance using the linear regression algorithm. 
For example, the quiz with related events (quiz attempt started, quiz attempt submitted, quiz attempt summary 
viewed, quiz attempt viewed) gave a high correlation, which means that the students’ performance was most 
affected with quiz. We have also studied the relationship between e-learning usage and student performance 
based on weeks’ activities using a linear regression algorithm. The result showed that the period between week 
nine and week 12 most affected students’ performance. 
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